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ABSTRACT 



In order to better understand the role of child-centered 



learning strategies in creating democratic, collaborative behaviors for 
states of the former Soviet Union, this study evaluated the impact of Step by 
Step, an early childhood development program in Bulgaria, Kyrgyzstan, 

Romania, and Ukraine. The study compared educational performance and 
developmental progress of preschoolers enrolled in the program with that of 
children in traditional kindergartens. The program's critical elements are 
family involvement, a child- centered curriculum, teachers as facilitators, 
and building collaborative relationship with important educational systems. 
Child assessment batteries were completed during May 1998 for 587 five- to 
seven-year-olds selected from 5 Step by Step programs and 5 traditional 
kindergartens in each country. Children were selected according to a 
stratified, randomized method with gender being the stratification criterion. 
Child assessment measures included the Test of Early Mathematics Ability; 
Emergent Literacy Assessment; Adapted Peabody Picture Vocabulary Test; and 
Torrance Tests of Creative Thinking (TTCT) . Findings for Bulgaria indicated 
that Step by Step children were performing at the same level as children in 
traditional programs. In Kyrgyzstan, Step by Step children dramatically 
outperformed those in traditional programs in both numeracy and literacy. In 
Romania, children in Step by Step classrooms from the low and high ends of 
the developmental spectrum were gaining more mathematical understanding than 
children in traditional programs and scored higher on the TTCT Unusual Uses 
test. In Ukraine, Step by Step children outperformed children in traditional 
programs in mathematics, the flexibility subscale of the TTCT, and receptive 
vocabulary. (KB) 
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CHAPTER I: OVERVIEW OF THE EVALUATION 



Purpose 



This program evaluation was funded in 1998 by the United States Agency for 
International Development (USAID), as part of the Improving Educational 
Quality Project (EEQ) II. The overarching purpose of the evaluation was to gain a 
better understanding of the role of child-centered learning strategies in creating 
democratic, collaborative behaviors at the local level for newly independent states 
of Eastern Europe and Central Asia. As such, it supports USAID/ENTs strategic 
goals of democratic transition and social stabilization by examining the role of 
participatory educational practices in promoting democratic behavior. 

The evaluation focused on Step by Step, an ongoing and growing early childhood 
development program in the ENI sector. Working with host country researchers in 
four countries — Bulgaria, Kyrgyzstan, Romania, and Ukraine — Education 
Development Center, Inc. (EDC) examined Step by Step’s impact on children, 
parents, and communities. First, we compared educational performance and 
developmental progress of preschool children enrolled in the Step by Step 
program with children in traditional programs. We also investigated the program’s 
effects on families, teachers, and school administrators. Third, using a range of 
methods, we examined Step by Step programs’ institutional capacity and uptake in 
key educational systems in order to gauge its sustainability. Finally, as with all 
IEQII initiatives, we sought to build the capacity in applied evaluation and 
assessment techniques within host countries. This paper reports only on the 
comparisons of educational performance of children in Step by Step compared to 
those in traditional programs. 

In this chapter, we describe the Step by Step model, discuss the importance of this 
evaluation and its implications for future policies, and briefly summarize the study 
methods and instruments. 
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Overview of the Step by Step Model 

Since its inception in 1994, the main objective of the Step by Step program has 
been to turn formerly state- supported day care facilities into centers for child 
development which promote learning and encourage democratic behaviors among 
children and their families. The Open Society Institute (OSI) funds the Step by 
Step program in the belief that educating young children in a manner that 
encourages family participation and individualized teaching, while supporting 
children’s ability to make choices, will lead to a new generation of citizens 
equipped to live in democratic societies. 1 The Step by Step model was developed 
by Children’s Resources International, Inc. (CRI) in partnership with OSI, and 
they continue to refine the model, support the expansion of the program, and 
create a forum for networking among Step by Step programs operating throughout 
the world. What began as an ambitious pilot in 15 countries with 250 classrooms 
is now an accepted educational program in 26 countries with more than 5,636 
classrooms implementing the Step by Step methodology. 

To better understand the evaluation and the relevance of the instruments used, it is 
important to have a general notion of the key features of the Step by Step 
methodology. The Step by Step model is based on four critical elements: 

• Family involvement, and parent participation in particular, is a mainstay of the 
Step by Step model. Families are encouraged to become actively involved in 
their children’s classrooms, working with teachers to implement the 
curriculum. Family involvement also extends beyond the classroom, as family 
members often contribute to the larger school community. 

• Child-centered curriculum is another key feature which stresses that teaching 
in Step by Step classrooms should be individualized, reflect the cultures and 
traditions of participating families, and provide opportunities for child choice 
and learning through play and experimentation. The classroom is, therefore, 
organized into activity centers to support the child-centered curriculum. 

• To support child-centered practices, the teacher’s role is transformed from 
transmitter of information to facilitator. This has implications for building 



1 P. Coughlin, presentation at International Conference on Early Childhood in Mongolia, 1997. 
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teachers’ understanding of child development and the ways in which they 
assess learning and development. 

• Effective program implementation depends on building collaborative 

relationships with important educational systems within countries, especially 
ministries of education and teacher training institutions. Such collaborative 
relationships need to be established at the kindergarten level as well — among 
administrators, teachers, and parents. 

OSI and CRI have worked together to introduce the program to participating 
countries. In collaboration with the local OSI foundation, they select a country 
team that will assume responsibility for launching the program; typically the 
country team consists of a country director and master teacher trainers. While all 
programs are required to incorporate the key features mentioned previously, 
country teams are encouraged to adapt the Step by Step model to fit the unique 
characteristics and traditions within each country. Such an approach is likely to be 
a contributing factor to the program’s growth. 

Since 1994, there has been an increasing demand for Step by Step and the 
enrollment figures are impressive. The program has not only expanded in size; but 
also in scope. The program now extends beyond preschool 2 to primary school, 
infant and toddler care, and orphanages. Over time, the Step by Step model has 
also formalized its outreach to institutions of higher education responsible for 
teacher training. CRI now conducts annual international seminars for faculty. 
These seminars provide course material on specific aspects of the Step by Step 
methodology, such as observing young children and learning though play. 

Over the past several years, there has been an increasing emphasis on making Step 
by Step programs self-supporting. Country teams have been encouraged to 
establish non-governmental organizations (NGOs), and seek administrative 
independence from local OSI foundations, which initially housed the program. As 



2 Preschools are referred to as kindergartens throughout the remainder of this report and typically 
include children between the ages of three and seven, though some kindergartens have historically 
provided services to infants and toddlers as well. 
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we launched this study, only Romania had NGO status. As the study has drawn to 
a close, Bulgaria and Kyrgyzstan have also established themselves as NGOs. 

As Step by Step moves toward organizational independence, the program has also 
undertaken a new initiative. Programs have worked to establish associations for 
parents, teachers, and early childhood faculty. The intention is that these 
associations will become active mechanisms within and across countries to 
advocate on behalf of children and to sustain the program’s philosophy. The move 
toward NGOs, and the increasing weight placed on these associations, signals that 
Step by Step may be here to stay — not a fleeting demonstration program. 

Increasing demand for the program, informal accounts of its impact, and the 
energy devoted to creating an infrastructure have suggested that Step by Step has 
been highly successful in reaching its goals. Host countries view the child- 
centered methodology and the emphasis on family participation as a powerful way 
to contribute to the development of democratic ideals and behaviors in children, 
parents, and the community. Prior to this USAID-funded evaluation, however, the 
impact of the program on children, families, and teachers, and its effect on local 
participation in schools has remained anecdotal and has not been studied 
systematically. Consequently, USAID has undertaken this rigorous evaluation to 
determine whether these child-centered practices lead to more democratic 
behaviors at the local level. 

Importance of Evaluating This Model 

Eastern Europe and Central Asia have had a long tradition of state-supported early 
childhood education. Even though these child care programs differed in their 
quality and scope, they could be found in most city neighborhoods and in many 
towns and villages. With the dissolution of the Soviet Union came social and 
economic upheaval, which had a profound effect on the social sector — particularly 
child care. One has only to review UNESCO data that compares attendance rates 
in preprimary programs from 1989 to 1996 to grasp the magnitude of the impact. 



3 CRI reports that Ukraine is currently in the final stages of the application process. 
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For example, in Bulgaria preprimary enrollment dropped significantly from 93 
percent in 1989 to 62 percent in 1996. 4 Even in newly independent states such as 
Kyrgyzstan, where attendance had not been as widespread, rates fell precipitously. 

The Step by Step program was introduced at a time when the child care system in 
this sector of the world was foundering, but still operational. Its child-centered 
methodology directly related to the goals of these newly independent states to 
build a democratic society. Its climate of openness that invited parent participation 
and encouraged teachers to make decisions matched the spirit of the revolutionary 
movement in progress. 

Change in well-entrenched, bureaucratic systems are slow and difficult to effect. 
Yet Step by Step appears to have galvanized the energies of diverse adults and 
propelled them into an organization that is implementing sweeping changes. By 
identifying the young child as the lever for change, parents, teachers, 
administrators, and local authorities seem to have been drawn into a movement 
that began with the education of young children but could, with time, influence 
and alter many social institutions. By concentrating its reform efforts on children 
in the earliest levels of schooling, the Step by Step model is strategically designed 
to leverage change by building momentum that will gather force as children move 
through the educational system. 

The importance of child-centered, participatory learning programs for promoting 
democratic behaviors among young children, and the emphasis on family and 
community participation, suggest an important learning opportunity within ENI 
countries about the role of education in fostering local empowerment. The 
program may offer a model for encouraging community initiative to meet social 
needs. The involvement of local research institutions with the program offers the 
possibility to create an institutional capacity for assessing the programs and 
incorporating the findings into pedagogical practice. 



4 UNICEF (1999). The State of the World's Children. Accessed from the World Wide Web on 23 February 
1998 at http://www.unicef.org/sowc99e.pdf 
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Methodology 



To evaluate the impact of Step by Step, we used multiple methods including a 
quasi-experimental design and qualitative approaches. Quasi-experimental 
methods were used to compare two types of programs — Step by Step and 
traditional kindergartens — on measures of program quality and children’s 
learning. 

During the initial start-up of the evaluation, EDC’s research team worked closely 
with the staff of CRI and the Step by Step leadership within each of the four host 
countries: Bulgaria, Kyrgyzstan, Romania, and Ukraine. We worked together to 
refine the study questions and gather preliminary data so that the methods and 
instruments used could effectively assess the program’s impact. We also recruited 
four highly qualified in-country research coordinators who served as essential 
collaborators throughout the entire research effort. We collaborated on the study 
design, instrument selection and adaptation; research coordinators also monitored 
the quality of data collection. 

We devised research methods and instruments to address the following research 
question: 

Are the educational performance and developmental progress of Step by Step 
children comparable to those of children in traditional programs? 

• How do children compare on mathematical, literacy, problem solving, and 
creative measures? 

Instrumentation and Data Collection 

EDC worked closely with CRI in the early stages of the project to refine the study 
design and to prepare draft instruments for the initial wave of data collection. 
Because kindergartens in all four countries close or alter their program content for 
the summer months, it was critical to finalize child assessment and collect these 
data by early June 1998. After examining more than 20 instruments, EDC 
identified three child assessment instruments that were appropriate assessments 
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for the constructs to be measured and could be adapted for use in all four 
countries. Since we were unable to locate a commercially available instrument to 
assess key components of early literacy, we adapted a tool we had developed for 
our ongoing program of domestic research in early childhood. 

While we shared our initial thinking about study instruments with research 
coordinators, the careful examination of possible instruments and the resulting 
decisions occurred in a cross-country research meeting Bucharest, Romania in 
April 1998. This face-to-face meeting and subsequent gatherings were important 
for establishing critical understandings among our international team of 
researchers. 

To ensure the validity of the child assessment instruments, each research 
coordinator conducted a pilot test of the instruments and its adaptations. The pilot 
involved assessing children in both Step by Step and traditional programs and 
evaluating the results in conjunction with the U.S. research team. Pilot test data 
led to adaptations of all child and classroom instruments, which were 
implemented and monitored by U.S. researchers. All tools were translated into 
Bulgarian, Kyrgyz, Romanian, Russian, and Ukrainian. 5 The Russian versions 
were used both in Ukraine and Kyrgyzstan where there are large Russian-speaking 
populations. Translated versions were reviewed by educational experts within 
each country to ensure that the constructs were accurately represented. 

In spring 1998, research coordinators hired data collectors to administer child 
assessments. Because participation in the study offered the chance to learn new 
applied research methods, research coordinators were able to recruit an unusually 
skilled cadre of data collectors for this effort, including faculty from prestigious 
universities. In Ukraine, for instance, several notable developmental psychologists 
served in this capacity; in Bulgaria, graduate students competed for data collector 
positions. 



5 OSI New York provided the necessary funds for these translations. 



ERIC 
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Although recruitment was highly successful, research coordinators in all four 
countries trained a greater number of candidates than actually would be needed. In 
this way researchers had the opportunity to select individuals who demonstrated 
the best understanding of the methods and who were able to reliably conduct the 
assessments. Data collector training was divided into two five-day segments the 
first of which was training for the child assessment instruments. Training was 
conducted onsite in each country by a team composed of a U.S researcher and the 
local research coordinator. In addition to a thorough review of conceptual 
underpinnings of the study design, our training involved using the instruments 
with children in actual classrooms. We also had extensive conversations about 
scoring decisions, which helped us identify which of the trainees were best 
qualified to carry out the data collection activities. 

While onsite in host countries, we also worked closely with local research 
coordinators to review procedures for random selection of child sample, set up 
systems for data management, and design mechanisms for quality control during 
data collection. 

Child and Classroom Data Collection and Analysis 

Data collection occurred primarily during May 1998 in five locations in each of 
the four countries. During this period child assessment batteries were completed 
on 587 children. Research coordinators traveled with data collectors and worked 
with them to resolve problems encountered, to monitor their efforts, and to review 
score sheets. Research coordinators also debriefed with data collectors as they 
returned their material. Score sheets were reviewed, inventoried, and sent to EDC 
for data entry. Once received in the U.S., data were checked, prepared for entry, 
double entered, and cleaned by EDC research assistants. 

We primarily used analysis of variance and analysis of covariance models to 
compare group means from Step by Step and traditional groups. Statistical 
significance and confidence intervals were calculated to assist with the process of 
statistical inference. Effect size measures were calculated (r| 2 and/or Cohen’s d) 
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when practical to assess the size and importance of the differences between 
groups. For categorical variables, contingency table analysis was used to compare 
percentages for the Step by Step groups and traditional groups. Pearson % tests 
were used to test statistical significance, and phi and Cramer’s phi statistics to 
measure effect size. Occasionally missing data reduced the Ns reported below, 
though missing data were minimal throughout the study. 

A second cross-country meeting was held in Budapest, Hungary in November 
1998 to discuss and interpret the preliminary results from the first phase of data 
collection. 

Limitations of Research Design 

Undertaking an international research effort of this scope in less than a 12-month 
period was a challenge. Because of the time and logistical constraints, our ability 
to answer the key question about the program’s impact on children is limited for 
three main reasons. First, we were only able to collect data at one point in time. It 
would have been preferable to collect data at two or more points in time to enable 
us to determine the amount of growth children made while in kindergarten. 
Second, we had very little information about children’s families, leaving open the 
possibility that children in Step by Step and traditional classrooms came from 
different kinds of homes. We attempted to control for this problem by drawing our 
two groups of children from the same schools and by identifying the child’s home 
language and their ethnicity. Also, we used a random selection procedure to 
ensure as much comparability as possible between the two groups. Nonetheless, 
there could be systematic biases in the selection of families whose children attend 
one type of kindergarten or another. Third, we had no tests with norms for the 
countries where we were collecting data; therefore we could not compare the 
developmental status of children we were testing with expected developmental 
levels for the country. 
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Study Sample 



City/Kindergarten Sample 

In order to select our sample, we gathered data from each country team regarding 
the location and characteristics of kindergartens that first adopted the Step by Step 
methodology when the program was introduced. In each country, there were 
between seven and ten kindergartens in the initial wave of funding in 1994 and 
1995 and we elected to examine five of these in each country. Each kindergarten 
had both initial Step by Step classrooms as well as expansion classrooms. 
Expansion classrooms were added after the first year of operation and received 
less resources for start-up than did initial Step by Step classrooms. To select the 
five cities and, therefore Step by Step kindergartens, we considered a number of 
factors including: the urbanicity of the city/town, its distance from the central 
office of the country team, geographic diversity, and presence of ethnic minorities 
in the population. 

The sample consisted of children from five Step by Step and five traditional 
kindergartens in each of the four countries (see Table LI below). Traditional 
kindergartens were selected from the same city and neighborhood, where possible, 
and were also matched according to the characteristics mentioned above. With the 
exception of one kindergarten in Ukraine, two initial classrooms and two 
expansion classrooms were selected from each of the five Step by Step 
kindergarten sites in each country. 6 Two classrooms were also selected from five 
traditional kindergartens in each country. Thus, children were selected from a total 
of 10 kindergartens (30 classrooms) per country or a total of 40 kindergartens and 
120 classrooms across countries. 



6 In one of the five cities in Ukraine, expansion classrooms had to be selected from a nearby Step by Step 
program that was closely matched to the initial kindergarten program. 
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Table 1.1: Kindergarten, Classroom, and Child Samples 



Per Country Overall 





Step by Step 


Traditional 


Step by Step 


Traditional 


Number of 
kindergartens 


5 (6) a 


5 


21 


20 


Number of classrooms 
per kindergarten 


4 

(2 initial; 2 expansion) 


2 


— 


— 


Total number of 
classrooms 


20 

(10 initial; 10 expansion) 


10 


80 

(40 initial; 40 expansion) 


40 


Total number of 
children 


70 b (83) 


70 (84) 


293 


294 



In order to be included in the sample, Step by Step kindergartens met the 
following criteria: 

• They contained at least two Step by Step initial classrooms, that is classrooms 
that have been using the Step by Step methodology since the inception of the 
program in the country. 

• More than 50 percent of the children in the initial classrooms participated in 
Step by Step for at least two years. 

• Children in initial Step by Step classrooms ranged in age from five to seven 
years old. 

• Teaching staff in initial Step by Step classrooms had been employing the Step 
by Step methodology for at least a two-year period. 

When possible, we selected Step by Step kindergartens that had at least two 
expansion classrooms serving children within the specified age range. When 
kindergartens in the sample contained more than two initial and/or expansion 
classrooms, we randomly selected those classrooms that were included in the 
study. 



a Numbers in parentheses indicate sample size in Ukraine. 

b Seven per classroom, on average. Children in expansion classrooms were not part of this study. 
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Child Sample 



The child sampling plan was to include 140 children from each of the four 
countries — 70 from initial Step by Step and 70 from traditional classrooms, for a 
total of 560 children. Because Ukraine included additional children in the 
sample — 83 children from Step by Step classrooms and 84 children from 
traditional classrooms — it brought the total to 587 children across all four 
countries. Children were selected according to a stratified, randomized method 
with gender being the criterion for stratification. This sample size was selected 
because it enabled us to detect moderate effect sizes, i.e., those that are 
educationally relevant. See Tables 1.2 and 1.3 for a presentation of child 
demographics. 



Table 1.2: Child Demographics — Gender 





Step by Step 
N 


% 


Traditional 

N 


% 


Male 


138 


48 


129 


44 


Female 


152 


52 


162 


56 


Total 


290 a 


100 


291 


100 



0 Relevant data not available for three children from each condition. 
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Table 1.3: Child Demographics — Age 





Step by Step 


Traditional 


Age (yrs) 


N 


% 


N 


% 


4 


12 


4 


21 


7 


5 


81 


28 


90 


31 


6 


147 


50 


145 


49 


7 


53 


18 


38 


13 


Total# of 
Children 


293 


100 


294 


100 


Mean Age 


5.82 


5.68 



Study Instruments 

To address the research questions, we used child assessments in the areas of 
numeracy, literacy, and creative thinking. A more detailed description of study 
instruments and their psychometric properties can be found in Appendix I. 
Instruments developed specifically for this study can be found in Appendix II. 

Child Assessments 

Test of Early Mathematics Ability, Second Edition (TEMA-2) 7 . The TEMA-2 is a 
commercially available instrument, specifically designed to assess the mathematical thinking 
skills of young children who may not yet be readers. It measures informal mathematics 
awareness (e.g., relative magnitude concepts, counting skills) and also assesses children’s 
abilities in formal mathematics (e.g., knowledge of conventions, number facts, calculation 
skill). 

Emergent Literacy Assessment (ELA). 8 EDC developed the ELA to assess various aspects 
of children’s progress in early literacy development in four areas: 1) Letter Identification, 2) 
Emergent Writing, 3) Early Reading, and 4) Print Concepts and Reading Comprehension. 

Adapted Peabody Picture Vocabulary Test, Third Edition (PPVT-III). 9 The PPVT-III 

was adapted to assess children’s receptive vocabulary. 



7 Ginsburg, H.P. & Baroody, A.J. (1990). Test of Early Mathematics Ability, Second Edition. Austin, TX: PRO- 
ED. 

8 The ELA was an adaptation of the Early Literacy Profile developed by David K. Dickinson and Carolyn 
Chaney, ©Education Development Center, Inc., 1998 with permission of the authors and publisher. 

9 Dunn, L.M. & Dunn, L.M. (1997). Peabody Picture Vocabulary Test, Third Edition. Circle Pines, MN: 
American Guidance Service, Inc. 
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Torrance Tests of Creative Thinking (TTCT). The TTCT is an instrument designed to 
assess the important characteristics of creative thinking on the dimensions of fluency, 
flexibility, and originality. Due to time constraints, we used only two activities of the verbal 
subtest of the TTCT: Product Improvement and Unusual Uses. 
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CHAPTER II: IMPACT 
ON CHILDREN’S LEARNING 



In this chapter, we will examine the progress of individual children in core 
academic areas. This child progress will be an important indicator that will help 
determine whether or not local decision makers and parents continue to support 
this approach to education. Step by Step recognizes this challenge and has 
responded by developing a pedagogical approach designed to teach children 
literacy and numeracy competencies along with other core knowledge valued by 
each country, in the context of classrooms that foster development of creativity, 
responsibility, and problem-solving skills. 10 

The individual child assessment part of this evaluation addressed the following 
research question: 

Are the educational performance and developmental progress of Step by Step 
children comparable to those of children in traditional programs? 

The assessment tools used were chosen to address the most salient indicators of 
child progress in the core academic areas of literacy, numeracy, and language 
development. In addition, we used a tool that is widely employed to assess 
creative thinking, because one of the special features of Step by Step is its effort to 
nurture children’s creative development. The characteristics of each instrument 
and their psychometric properties are discussed in more detail in Appendix I. 

All of our child assessments were built on two core assumptions: 1) our tasks 
elicited performances that reveal information central to understanding a child’s 
developmental level in the domain being assessed, and 2) our tasks allowed us to 
chart the developmental trajectory of children as they acquire skill in a given 
domain. That is, we expected that older children would do better than younger 

10 In our on-site visits to Step by Step programs we found that topics related to the history, culture, and artistic and 
musical traditions of the country are also an important feature of Step by Step kindergartens. It was evident that to 
some degree, a major criterion guiding selection of this material was a desire to expose children to knowledge 
common to all children in the country. It is well beyond the scope of this study to assess children’s mastery of this 
country- and culture-specific knowledge. 
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children; therefore performance differences would reveal developmental 
differences in a domain. We also assumed that highly effective programs would 
move children through these developmental progressions more quickly than less 
effective programs. Given these assumptions, it was essential that we take age into 
account when comparing children from different programs. One way to control for 
age is to ensure that all children are the same age. Unfortunately, the samples 
drawn from Step by Step and traditional programs were not consistently matched 
in age. We therefore used statistical methods to adjust for age in our means. As a 
result, all mean scores and all analyses comparing children in Step by Step to 
those in traditional programs take into account the effect of age differences among 
children in the sample for a given country. 

In this chapter we make within-country comparisons, but no cross-country 
comparisons. We have taken this approach because the issue of interest is whether 
children in Step by Step programs are achieving at levels consistent with those 
typically seen in a given country. The success of Step by Step must be viewed in 
terms of its ability to use the resources and organizational systems of a given 
country to educate. Even if one wanted to compare across countries, there would 
be no psychometric basis for making cross-country comparisons using the tests we 
employed. Therefore, while the display of results may appear to invite 
comparisons among countries, it is not valid to do so. 

In this chapter, we drew our sample of Step by Step children only from initial 
classrooms. Consequently, we present findings only from children in Step by Step 
initial classrooms and the comparison group in traditional classrooms. 

Numeracy 

The Test of Early Mathematical Ability, Second Edition (TEMA-2), was used to 
evaluate both formal and informal mathematical knowledge. The informal skills 
measured include relative magnitude, counting, and calculation skills. These are 
the kinds of skills that children can acquire as they play with materials, participate 
in games, and engage in casual conversations with adults about their activity. 
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They might acquire such knowledge at home or during the course of informal 
kindergarten activities and conversations. The formal abilities measured include 
knowledge of convention, number facts, calculation skills, and base ten concepts. 
Children are more likely to acquire these capacities through direct instruction; 
thus they are more likely to acquire them in school. In the test, more informal 
items appeared earlier, and formal items later. However, even five- and six-year- 
old children encountered some of both kinds of items. 

Because the score a child receives reflects performance on a composite set of 
items, it is difficult to interpret raw scores. In the U.S. raw scores are given 
meaning by relating them to age norms. No such norms exist for the countries 
included in this study, but it is reasonable to expect that children in other countries 
acquire the mathematical understandings tapped by the TEMA-2 in roughly the 
same order as U.S. children. Because we use age-adjusted means and lack 
country-specific norms, one cannot use these U.S. mean scores to draw 
conclusions about the rate of acquisition of mathematical knowledge in other 
countries. They do, however, provide a starting point for considering cross- 
national developmental pathways in mathematics learning. Therefore, Table IV. 1 
displays patterns of performance reflected by different total scores. These allow 
the reader to attach some meaning to the raw scores. The points along the 
continuum describing score profiles correspond to the score achieved by an 
average U.S. child between the ages of four and eight. It should be noted that the 
linkage between these average profiles and any given child’ s actual performance 
is only approximate, because children will vary in the items they pass and those 
they miss as they approach a ceiling. Nonetheless, this chart provides some 
guidance regarding overall patterns of development. 
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Table IV.l: TEMA-2 Profiles 



!'*/■ Score ' 


’ ‘ : I~ 1; ' •« Performance Profile ■■ ■ ■ %W:'- ■ . : 

^ *- : A. . -M ^ - ^ ; «• • < 


13 

U.S.: 4 year old 
performance 


Informal 

Counting. Counts up to 10 objects; enumerates up to 5 objects 

Formal 

Conventions. Reads single-digit numerals 


23 

U.S.: 5 year old 
performance 


Informal 

Counting. Forward in the 20’s and 30’s; backwards from 10 to 1; 
enumerates 9 and 10 dots on a card 

Calculation. Up to 2 + 5 pennies being combined in a hand 

Relative magnitude. Single-digit proximity on a number line (e.g., 5: closer 

to 1 or 7?; 3: closer to 1 or 6?) 


31 

U.S.: 6 year old 
performance 


Informal 

Counting. Forward to 42; by 10’s until 90 

Relative magnitude. Double-digit proximity on a number line (e.g., 32: 
closer to 24 or 61?) 

Formal 

Conventions. Writes numbers in the teens; reads double-digit numbers 


39 

U.S.: 7 year old 
performance 


Informal 

Counting. Backwards starting at 20; by tens up to 160 

Formal 

Number facts. Rapid recall of facts (2 + 2; 3 + 4; 6 + 3) 
Conventions. Writes three-digit numerals 


46 

U.S.: 8 year old 
performance 


Informal 

Counting. Continues sequences over 100 started by examiner (e.g., “161, 
162,...”) 

Formal 

Number facts. Addition facts (8 + 8,7 + 7); subtraction (8 - 4, 12 - 6) 
Base Ten. Number of $10 bills in $100? $100 bills in $1000? (local 
currency was used) 

Conventions. Writes three-digit numerals 
Calculation. Adding without carrying (23 + 15, 64 + 32) 
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Bulgaria 



In Bulgaria there were no significant differences between the performance levels 
of children in the Step by Step program (Mean = 25.9, SD = 9.5) and children in 
traditional programs (Mean = 28.01, SD = 10.42). These overall mean scores 
indicated that children are performing at similar levels and acquiring skills such as 
counting in the twenties, enumerating 10 objects, and using a number line to 
determine the relative proximity of a single-digit numeral (e.g., 5) to two others 
(e.g., 1, 7). While differences between the two types of kindergartens were not 
statistically significant, inspection of graphs depicting the frequency data (see 
Figures IV. la and IV. lb), revealed a somewhat greater incidence of scores at the 
lower end of the continuum among children from Step by Step classrooms. Of all 
children assessed in Step by Step classrooms, 45 percent scored 22 points or less; 
whereas 30 percent of the children from traditional programs received such scores. 

Figure IV.la: TEMA-2 
Bulgaria: Step by Step (n=70) 
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Figure IV.lb: TEMA-2 
Bulgaria: Traditional (n=70) 




Children’s Raw Scores 





Education Development Center, Inc. 



23 



24 



Kyrgyzstan 



In Kyrgyzstan, the mathematical achievement of children in Step by Step 
programs (Mean = 32.30, SD = 7.88) was significantly higher (p < .001) than that 
of children in traditional kindergartens (Mean = 26; SD = 8.67). The size of these 
differences was dramatic, as indicated by the very large effect size of .761. An 
effect size of this magnitude is very rare in educational research and indicates that 
the difference observed is not only statistically significant, but is of considerable 
educational importance. Another way to consider these differences is in 
developmental terms. In the United States the point differential observed between 
the two programs (8.3 points) is roughly equal to the point difference seen 
between children of two different ages (7.9 points between five and six, 8.2 points 
between six and seven). 

If we examine Table IV. 1 , we can get a qualitative sense of the performance levels 
of children in these two types of kindergartens. Children in Step by Step programs 
could count into the forties and by tens and demonstrate varying skills with 
double-digit numbers — reading and writing, determining their relative 
magnitude — while, in general, children in traditional programs demonstrated skills 
with single-digit numbers and could count in the twenties. 

Figures IV.2a and IV. 2b provide yet another way to understand the differences 
between children in these two programs. Far more children in traditional 
kindergartens received very low scores, with 23 percent of the sample receiving 
scores of 15 and below; no child in Step by Step received such a score. Similarly 
dramatic differences were apparent at the high end of the continuum, with 34 
percent of the children in Step by Step receiving scores of 40 and above, 
compared with only 4 percent of the children in traditional kindergartens. In 
essence the Step by Step program appears to have resulted in considerable gains 
across the entire developmental spectrum, boosting achievement of children at the 
lower end and raising that of more able children. 
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Figure IV.2a: TEMA-2 
Kyrgyzstan: Step by Step (n=70) 
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Figure IV.2b: TEMA-2 
Kyrgyzstan: Traditional (n=69) 
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Romania 



In Romania the performance of children in Step by Step programs (Mean = 28.18, 
SD = 8.15) was significantly stronger (p < .05) than that of children in traditional 
kindergartens (Mean = 25.63, SD = 7.13). The effect size of .334 indicated 
moderate differences that are large enough to be of educational importance. As 
indicated in the figure describing performance levels, we see that children in both 
types of programs are gaining skill counting in the twenties. In addition, children 
in both types of programs are beginning to know simple addition facts and have a 
sense of the relative magnitude of single-digit numbers. More advanced children 
are gaining skill reading and writing numerals. 

Graphs of scores presented in Figures IV. 3a and IV. 3b reveal interesting 
differences at both ends of the developmental continuum. Scores of 23 or less 
were received by 44 percent of the children from traditional programs but by only 
23 percent of the Step by Step children. Conversely, scores of 35 and higher were 
received by 24 percent of children from Step by Step programs and only 13 
percent of the children from traditional programs. Thus, in comparison to the 
traditional kindergartens, the Step by Step program appears to be somewhat more 
successful in raising the performance of children with less advanced 
understanding of mathematics while also supporting growth of more advanced 
students. 
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Figure IV.3a: TEMA-2 
Romania: Step by Step (n=70) 
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Figure IV.3b: TEMA-2 
Romania: Traditional (n=70) 
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Ukraine 



In Ukraine, Step by Step children’s mathematical achievement was significantly 
higher than that of children in traditional programs (p < .001). The difference 
between scores of children in Step by Step kindergartens (Mean = 34.96; SD = 
10.79) and those in traditional kindergartens (Mean = 28.92; SD = 8.98) was very 
large, as indicated by the effect size of .61 1, an effect size that is rarely seen in 
educational research. From Table IV. 1 one can note the differences between 
average performances of Step by Step children and those from traditional 
programs. On average, Step by Step children displayed a more advanced sense of 
the relative magnitude of double-digit numbers and have made more progress in 
acquiring conventional mathematical skills related to knowledge of number facts 
and reading and writing double-digit numbers. 

The graphs displaying frequencies of different scores (Figures IV.4a and IV.4b) 
reveal the impact the Step by Step program made at both ends of the 
developmental continuum. Relatively low scores of 26 and lower were three times 
as frequent for children in traditional classrooms (37 percent) as for children from 
Step by Step classrooms (13 percent). Conversely, scores of 41 and higher, at the 
top of the developmental continuum, were nearly three times more frequent 
among children from Step by Step programs (31 percent) than among children 
from traditional programs (11 percent). Thus, the Step by Step program is more 
successful in supporting development of children across the developmental 
continuum. 
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Figure IV.4a: TEMA-2 
Ukraine: Step by Step (n=83) 




Figure IV.4b: TEMA-2 
Ukraine: Traditional (n=84) 
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Literacy 



A central objective of schools that serve children between the ages of five and 
eight is to help them learn to read and write. Research in the U.S. has established 
the fact that children’s early literacy development is a strong predictor of future 
academic performance. Literacy learning has also been a high national priority in 
all four countries we studied. Therefore, it is important that Step by Step programs 
support children’s literacy development during these critical early years. 
Considerable research by developmental and cognitive psychologists has 
established that literacy development, especially during the early years, is 
multifaceted. While reading and writing appear to be distinct skills, both draw on 
a common core of knowledge that children are constructing. This core of 
knowledge includes oral language in addition to more typical literacy-related 
knowledge (e.g., letter recognition, decoding, spelling). Our task, therefore, was to 
assess children’s early literacy development broadly. To this end we used a broad- 
gauged assessment of early literacy, the Emergent Literacy Assessment (ELA) 
and a receptive vocabulary test. The ELA is divided into four subtasks: Letter 
Identification, Emergent Writing, Early Reading, and Print Concepts and Reading 
Comprehension. The receptive vocabulary tool was an adaptation of the Peabody 
Picture Vocabulary Test, Third Edition (PPVT-III). We worked closely with team 
members as we devised tools appropriate for local circumstances. 

To provide the reader with a more qualitative sense of the results from each 
country, Table IV. 2 describes what each ELA subtest involves and the number of 
points required to receive full credit. For each subtest we have clustered scores 
into rough groupings and have indicated the general developmental level 
associated with each cluster. These clusters are provided simply to help the reader 
conceptualize the relative developmental level of the “average” child. 
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Table IV.2: Contents of the 
Emergent Literacy Assessment (ELA) 



Subscale 

OVERALL TEST 


Scoring 

TOTAL poss.: 76 pts. 


Prompts 


Range 


Letter Identification 
Subscale 


total possible: 24 pts. 






( 8 upper case, 4 lower 
case letters ) 


2 points for naming letters and 
1 point for pointing to letter 
named by assessor 


“What letter is this?” 
(letter naming) 

“Can you show me the 
B?” 

(letter identification) 


4-8: Emergent 

beginning to note letters of special interest 
9-14: Early Mastery 

familiar with several letters, name some on 
demand 

15-22 : Advanced 

identifies and names many letters 

23-24 : Expert 

solid letter knowledge 


Emergent Writing 
Subscale 


total possible: 22 pts . 






Name Writing 

(writes own name) 

Own Word Writing 

(writes 2 words of own 
choosing) 

Spelling 

( writes 3 words ) 


total possible: 4 pis. 
intends to write name, correct 
letters used 
total possible: 9 pts. 
directionality (2 pts.), grasp (1 
pt.), spelling accuracy (9 pts). 
total possible: 9 pts. 
spelling accuracy 


“Show me how you write 
your name.” 

“What did you write?” 

“Show me how you 
wrote it.” 

“Please write ... Spell it 
the best way you can.” 


2—5: Early Emergent 

early sense of how writing is organized and what 
is expected when one is asked to write 
6-8 : Emergent 

knows some conventional forms associated with 

familiar words 

9-1 2: Early Conventional 

associating initial sounds to symbols, some known 
words 

13-18: Advanced 

associating final and some medial sounds to 

symbols 

19-22: Expert 

grasp of sound-symbol correspondence for words 
in early writing repertoire 


Early Reading Subscale 


total possible: 9 pts. 






( reads own name and 8 
more words of increasing 
difficulty) 




“Can you tell me what 
this says?” 


1: Emergent 
can only read own name 
2-4: Early Conventional 
some sight words 
5-7: Advanced 

associating intial sounds to symbols; early 
decoding skill 
8-9: Expert 

grasp of how to approach decoding; skill with 
initial reading vocabulary 


Print Concepts and 
Reading Comprehension 
Subscale 


total possible: 21 pts. 






Book & Print Concepts 
Comprehension 


total possible: 10 pts. 

Book concepts (4 pts.) 
(handling, author) 
Directionality (2 pts.) 

Word/letter concept (2 pts.) 
Written-spoken word 
matching (2 pts.) 
total possible: 1 1 pts. 
literal recall (5), inferential (6) 


How book is held. 
“What did (author) do?” 
Which way do you go 
when reading? 

“Show me a word.” 

Read and finger point 
“What happened first? 
Second?” 

“What do you think is 
going to happen?” 


4-8: Ea rly A wa ren ess 

Some sense of book use, limited understanding of 
story line 

9-13: Developing Reader 

General sense of book use and print conventions; 

able to follow major thread of simple story line 

14-18: Experienced Reader 

Grasps basic print concepts; follows story line 

including most inferences 

19-22 : Veteran Reader 

Solid grasp of print concepts; skilled at recalling 
details and inferring critical information 
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Bulgaria 



Children in Step by Step and traditional programs scored at comparable levels in 
acquisition of literacy skills. Receptive vocabulary scores were not significantly 
different, and on the ELA, children’s overall and subtest scores were also 
comparable. Table IV. 3 shows that, on average, children in both settings 
demonstrated “advanced” letter identification skill, indicating that most children 
could name several letters and point to others. The writing mean scores of 14 
placed children of both groups at the low end of the “advanced” grouping, 
reflecting ability to write some familiar words and beginning ability to sound out 
others. The word reading scores of 4 placed children near the top of the “early 
conventional” band, indicating that children are beginning to read some familiar 
words and that some are associating some sounds to letters. Children’s scores on 
the Print Concepts and Reading Comprehension subscale suggested that, on 
average, children have a basic grasp of how books work and are developing some 
skill in understanding the vocabulary and following the plot line of a story. Thus, 
children in both programs are beginning to acquire knowledge required for 
reading and writing, and are just beginning to solidify basic understanding 
required for conventional reading and writing. 



Table IV.3: Literacy Findings — Bulgaria 



Step by Step 





Adj. Mean 


SD 


n 


ELA 


Letter 

Identification 


17.9 


6.99 


69 


Emergent Writing 


14.33 


7.26 


69 


Early Reading 


3.95 


2.97 


69 


Print Concepts 
and Reading 
Comprehension 


15.16 


3.61 


69 


Total 


51.76 


18.58 


69 


PPVT 


83.40 


27.73 


70 



a Not statistically significant 



Traditional 



Adj. Mean 


SD 


n 


16.76 


7.70 


70 


13.66 


8.58 


70 


4.11 


3.25 


70 


14.68 


4.94 


70 


49.65 


21.72 


70 


86.23 


32.13 


70 
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While differences between Step by Step and traditional kindergartens are not 

significant overall, frequency graphs suggest some potentially interesting 

differences in how these two kinds of programs support children’s growth (see 

Figures IV. 5a and IV. 5b). Interestingly, different patterns appeared at the two 

extremes. If we look at children who received overall scores of less than 20 (a 

> 

total score reflecting very limited grasp of literacy conventions), we find 14 
children in traditional programs with such scores as opposed to only 6 Step by 
Step children. It seems that Step by Step may be somewhat more able to support 
the growth of children who are having some difficulty acquiring basic literacy 
skills. On the other hand, if we examine the number of children receiving scores 
of 54 or above, we find both programs to be comparable. These results are based 
on only a small number of children, but they suggest that both types of programs 
are supporting literacy development of many children. However, traditional 
programs may have difficulty meeting the needs of children in need of special 
support. 

Figure IV.5a: ELA 
Bulgaria: Step by Step (n=70) 
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Figure IV.5b: ELA 
Bulgaria: Traditional (n=70) 
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Kyrgyzstan 

In Kyrgyzstan, Step by Step kindergartens have had considerably more success 
than traditional kindergartens in supporting children’s literacy development. 
Dramatic differences were seen on children’s receptive vocabulary scores 
(adapted PPVT), with children in Step by Step programs receiving raw scores that 
averaged 20 points higher than children in traditional programs. Similar dramatic 
differences were evident on the ELA, with Step by Step children’s average scores 
(Mean = 60.37; SD = 15.76) far surpassing the total mean scores received by 
children in traditional programs (Mean = 40.32; SD 26.23). A quantitative 
expression of the difference is the enormous effect size of .955. 

If we examine the ELA subscales, we can understand the meaning of these 
differences in mean scores. On the Letter Identification subscale, children in Step 
by Step were solidly in the “advanced” score band, whereas children in traditional 
kindergartens fell into the “early mastery” band. Similarly, on the Emergent 
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Writing subscale Step by Step children displayed “advanced” skills, whereas 
children in traditional programs were just beginning to construct conventional 
understandings of print. Similarly, on the Early Reading and Print Concepts and 
Reading Comprehension subscales, Step by Step children were a full score band 
above children in the traditional kindergartens. Thus, across all dimensions of 
early literacy, children in Step by Step programs were significantly ahead of 
children in traditional programs in literacy and language development. 

Examination of frequency data in Figures IV. 6a and IV. 6b sheds light on the 
source of these enormous differences between the two kindergarten models. 
Thirty-two children in the traditional kindergarten sample (46 percent) received 
low total scores which fell below 20. In Step by Step classrooms only three 
children, 4 percent of the sample, received such low scores. Conversely, strong 
scores of 61 or higher were received by only 19 children in traditional classrooms 
(27 percent of the sample), but by 44 Step by Step children (63 percent). The 
bimodal distribution seen among children from traditional programs suggests that 
some children, perhaps those who enter school with home support for literacy, 
have benefited from the traditional approach employed, whereas many other 
children have made limited progress. 
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Table IV.4: Literacy Findings — Kyrgyzstan 



Step by Step Traditional 





Adj. Mean 


SD 


n 


ELA 


Letter 

Identification 


19.44 


6.79 


69 


Emergent 

Writing 


16.08 


6.40 


69 


Early Reading 


6.83 


2.94 


69 


Print Concepts 
and Reading 
Comprehension 


17.14 


3.65 


69 


Total 


60.37 


15.76 


69 


PPVT (Adapted) 


91.61 


30.03 


70 



Statistical significance: 

(a) pc. 001 

(b) pc. 0001 




Adj. Mean 


SD 


n 


13.18 


8.80 


70 


9.48 


8.94 


70 


3.72 


3.66 


70 


12.35 


4.35 


70 


40.32 


26.23 


70 


71.44 


31.56 


70 



Figure IV.6a: ELA 
Kyrgyzstan: Step by Step (n=70) 
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Figure IV.6b: ELA 
Kyrgyzstan: Traditional (n=69) 
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Romania 



Children in Step by Step and traditional kindergartens in Romania are making 
comparable progress toward acquiring early literacy skills. Mean scores on the 
receptive vocabulary test (adapted PPVT) were nearly the same (Mean = 76.6; SD 
= 77.2) and total ELA scores were similar (Mean = 40.7; SD = 37.4). Examination 
of subscales revealed comparable scores for children in both programs. On Letter 
Identification, the average scores placed children in the “early mastery” score 
band, indicating beginning ability to identify and name a few letters. The 
Emergent Writing scores of children in both programs placed them at the upper 
end of the “early conventional” band, indicating that, on average, children knew 
some words and were beginning to associate initial sounds of words with symbols. 
Early Reading scores also placed children in the “early conventional” score band, 
indicating that children could read their own name and were just beginning to be 
able to read other words. Print Concepts and Reading Comprehension results also 
placed children at the “developing reader” level, suggesting that they were 
beginning to understand how to handle books and how to follow major aspects of 
the story line of a book. 
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Table IV.3: Literacy Findings — Romania 



Step by Step 





Adj. Mean 


SD 


n 


ELA 


Letter 

Identification 


12.91 


7.18 


70 


Emergent 

Writing 


12.25 


7.41 


70 


Early Reading 


2.80 


3.14 


70 


Print Concepts 
and Reading 
Comprehension 


12.70 


3.64 


70 


Total 


40.66 


18.53 


70 


PPVT (Adapted) 


76.63 


19.57 


70 



Traditional 



Adj. Mean 


SD 


n 


11.63 


7.15 


70 


11.44 


7.67 


70 


2.22 


2.97 


70 


12.36 


3.32 


70 


37.38 


18.56 


70 


77.23 


14.58 


70 




“Not statistically significant 



Examination of frequency data (see Figures IV.7a and IV.7b) revealed trends that 
suggest that the overall impact of these two approaches to education may be 
having differential effects that do not appear when overall means are compared. 
Scores below 26 were somewhat more frequent among children in traditional 
kindergartens (37 percent of the sample) than among Step by Step children (26 
percent of the sample). Also, higher scores of 57 and greater were somewhat less 
frequent among children attending traditional kindergartens (19 percent of the 
sample) than among children in Step by Step classrooms (27 percent). It is likely 
that these differences in the extreme scores did not result in overall statistically 
significant differences, because children in the middle range showed somewhat 
stronger performance in traditional programs. Thus, it seems that Step by Step 
programs help support growth of less well-prepared children and may be 
nourishing somewhat more rapid development of children who start with a 
stronger foundation. 
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Figure IV.7a: ELA 
Romania: Step by Step (n=70) 
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Figure IV.7b: ELA 
Romania: Traditional (n=70) 
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Ukraine 



Children in Ukraine, regardless of program type, demonstrated relatively high 
mean scores (Mean = 63.01; SD = 60.69). There were no statistically significant 
differences in ELA scores, but raw receptive vocabulary scores of Step by Step 
children were more than 10 points higher than those of children in traditional 
kindergartens, a difference that is statistically significant. 

Review of the subscale scores revealed that, on average, children in both types of 
programs were quite skilled at Letter Identification. Emergent Writing and Early 
Reading scores were also strong, with children in traditional programs falling in 
the middle of the “advanced” score band and children in Step by Step programs 
appearing toward the upper end of this score band. Scores on the Print Concepts 
and Reading Comprehension subscale were low relative to the scores on the other 
subscales, with children in both programs at the low end of the “experienced 
reader” band. It is worth noting that children in both programs approached the 
ceiling score of 24. This result suggests that a larger sample or an assessment 
battery with more challenging items might reveal more information about 
children’s literacy development. 



Table IV.4: Literacy Findings — Ukraine 
Step by Step Traditional 



Adj. Mean 


SD 


n 


22.32 


3.19 


84 


16.93 


5.78 


84 


7.08 


2.89 


84 


14.28 


3.73 


84 


60.69 


13.01 


84 


90.76 


20.27 


84 






Adj. Mean 


SD 


n 


ELA 


Letter 

Identification 


22.03 


2.75 


83 


Emergent 

Writing 


18.18 


3.78 


83 


Early Reading 


7.65 


2.05 


83 


Print Concepts 
and Reading 
Comprehension 


15.10 


3.36 


83 


Total 


63.01 


9.76 


83 


PPVT 


101.37 


23.40 


83 



a Not statistically significant 

b p < .0083 i BEST COPY AVAILABLE 
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Frequency data in Figures IV. 8a and IV. 8b revealed an interesting pattern which 
possibly points to differential effects of the two approaches to literacy. First, both 
types of kindergartens appeared to be providing basic support to all children, since 
there were no extremely low scores (i.e., nothing below 21), and very few scores 
below 35. That said, it appeared that Step by Step may be somewhat more 
successful in ensuring that all children are making strong literacy progress. A 
comparison of the number of children with scores of 64 and below revealed that 
37 percent of the children in traditional programs received such “average” scores; 
whereas only 19 percent of the children in Step by Step classrooms received such 
scores. 



Figure IV.8a: ELA 
Ukraine: Step by Step (n=83) 




Children's Totals 
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Figure IV.8b: ELA 
Ukraine: Traditional (n=84) 




Children's Totals 



Creativity 

One of the distinctive features of the Step by Step approach is its emphasis on 
encouraging children’s creative problem solving. Although research on creativity 
has been done, it is extremely rare to include measures of it in a program 
evaluation. This is because we do not know the extent to which such measures are 
sensitive to performance differences that are affected by children’s classroom 
experiences. Despite this methodological challenge, we included the best available 
measures of creativity in our child assessment battery. We used two subtests of the 
Torrance Tests of Creative Thinking; Product Improvement and Unusual Uses. 
Each of these tasks were scored using three scales: fluency, flexibility, and 
originality. Fluency assesses children’s ability to generate several responses when 
asked, for example, what they might do with an empty box. Flexibility refers to 
children’s ability to generate responses from a number of different categories (i.e., 
“candy store, airplane, telephone booth” vs. “candy store, grocery store, clothing 
store”). Originality refers to the child’s ability to produce responses that are 



42 



Education Development Center, Inc. 



43 BEST COPY AVAILABLE 



deemed by the test constructors to be “unusual.” For a response to be considered 
“unusual” it must not appear on a list of “typical” responses. Thus, across the two 
tasks we obtained scores on these three dimensions of verbal creativity for each 
child. 



Bulgaria 



Among children in Bulgaria, we found no significant differences between the 
children on any dimension for either of the subtests (see Table IV. 5). However, 
the data did indicate an advantage of Step by Step children over children in 
traditional programs on the Product Improvement subtest. On all other 
dimensions, on each task children from both programs performed equally well. 



Table IV.5: Torrance — Bulgaria 
Step by Step Traditional 




Adj. Mean 


SD 


6.48 


5.04 


3.31 


2.41 


3.87 


3.65 


5.80 


5.01 


3.93 


2.57 


3.16 


3.82 





Adj. Mean 


SD 


Product 

Improvement 

Fluency 


7.08 


4.48 


Flexibility 


3.12 


1.82 


Originality 


4.54 


3.27 


Unusual Uses 






Fluency 


5.78 


5.77 


Flexibility 


4.16 


3.24 


Originality 


3.03 


4.14 



a Not statistically significant 



BEST COPY AVAILABLE 



Education Development Center, Inc. 



43 



Kyrgyzstan 



In Kyrgyzstan there were no statistically significant differences between children 
from Step by Step kindergartens and traditional kindergartens (see Table IV. 6). 
While there were no significant differences, on every subscale children from Step 
by Step scored slightly higher than children in traditional programs. This trend 
was especially apparent in the fluency and flexibility items on both activities. 



Table IV.6: Torrance — Kyrgyzstan 
Step by Step Traditional 



Mean 


SD 


3.77 


2.16 


2.53 


1.42 


.87 


1.05 


4.56 


2.05 


2.95 


1.30 


1.27 


1.17 





Mean 


SD 


Product 

Improvement 

Fluency 


4.43 


2.36 


Flexibility 


2.91 


1.54 


Originality 


1.13 


1.49 


Unusual Uses 






Fluency 


5.67 


3.74 


Flexibility 


3.61 


2.08 


Originality 


1.54 


1.72 



a Not statistically significant 
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Romania 



In Romania there was some evidence that children in Step by Step programs were 
able to be more creative than children in traditional kindergartens. On the 
flexibility subscale of the Unusual Uses task, Step by Step children performed 
significantly better (p < .001) than children from traditional classrooms, with the 
magnitude of these differences indicated by the educationally important effect size 
of .432. In addition, it is interesting to note that the mean showed a trend toward 
greater fluency and creativity in Step by Step children in the Unusual Uses task 
and for the flexibility subscale in the Product Improvement task. 



Table IV.7: Torrance — Romania 
Step by Step Traditional 



Adj. Mean 


SD 


5.06 


3.39 


2.62 


1.42 


2.3 


2.29 


5.01 


3.28 


3.08 


1.77 


2.06 


2.25 






Adj. Mean 


SD 


Product 

Improvement 

Fluency 


5.40 


3.83 


Flexibility 


3.08 


1.80 


Originality 


2.71 


2.48 


Unusual Uses 






Fluency 


6.11 


4.95 


Flexibility 


4.06 


2.77 


Originality 


2.81 


3.29 



a Not statistically significant 
b p<.01 
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Ukraine 



Children’s scores on the Unusual Uses subtest provided some evidence that 
children in Step by Step classrooms are more creative than children in traditional 
programs. On their flexibility scores, Step by Step children scored significantly 
higher (p < .05) than traditional children, with this difference associated with a 
moderate effect size of .354. While there were no other statistically significant 
differences, on the other two subscales for this activity children from the Step by 
Step classrooms received slightly higher scores on average than did children from 
traditional programs. 



Table IV.8: Torrance — Ukraine 
Step by Step Traditional 



Adj. Mean 


SD 


3.45 


2.91 


2.05 


2.39 


2.27 


2.47 


4.68 


3.65 


1.54 


2.39 


3.12 


3.08 






Adj. Mean 


SD 


Product 

Improvement 

Fluency 


3.39 


2.10 


Flexibility 


2.00 


1.62 


Originality 


2.24 


1.71 


Unusual Uses 






Fluency 


5.74 


4.53 


Flexibility 


2.25 


1.62 


Originality 


3.96 


3.31 



a Not statistically significant 
b p < .05 
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Summary 



Findings for Bulgaria 

Across all of the child assessment tasks, children in the Step by Step program 
seem to be performing at the same level as children in traditional programs. 
However, examination of trends revealed several hints suggesting areas worthy of 
further investigation. When comparing children with relatively low scores and 
ELA from the two types of kindergartens, we found a somewhat higher frequency 
of low scores among children from Step by Step children on TEMA-2. In contrast, 
we found that there were more low scores among children in traditional programs 
on the ELA. These results suggest the possibility that, when working with children 
who come to school less prepared for academic learning. Step by Step teachers 
may be more effective in supporting early literacy than early numeracy. 

Findings for Kyrgyzstan 

There were dramatic differences between children in Step by Step and those in 
traditional programs on all measures of academic achievement (i.e., numeracy and 
literacy). Results in mathematics showed an extraordinary effect size of .761, a 
magnitude that indicates a difference of considerable educational importance. 
Literacy results were equally impressive, with adapted PPVT results and scores on 
each of the ELA subscales being significantly higher for Step by Step children. 

Findings for Romania 

In Romania, there was evidence that children in the Step by Step kindergartens 
were making better progress than children in traditional programs. In mathematics 
we found significant differences associated with moderate effect sizes. Children in 
Step by Step classrooms from both the low and high ends of the developmental 
spectrum seemed to be gaining more mathematical understanding than were 
children in traditional programs. While differences between programs on the ELA 
were not statistically significant, there was evidence that Step by Step classrooms 
may be somewhat more successful in supporting the development of children with 
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the weakest early literacy skills. Finally, there was clear evidence of Step by Step 
children’s stronger performance on the flexibility scale of the Unusual Uses task, 
suggesting that these classrooms are nurturing children’s creativity more 
effectively. 

Findings for Ukraine 

In Ukraine, Step by Step children outperformed children in traditional programs in 
mathematics. The differences were highly significant and showed an effect size of 
.61 1, a magnitude of considerable educational importance. Significant differences 
favoring Step by Step were also seen on our measure of receptive vocabulary. 
However, children in Step by Step and traditional programs received similar 
scores on other measures of early literacy. Finally, there was also evidence that 
Step by Step settings are more effectively supporting children’s verbal creativity 
as children showed stronger performance on the flexibility subscale of the 
Unusual Uses subtest. 

Conclusions and Discussion 

We can answer the question of the Step by Step children’s educational 
performance being comparable to that of children in traditional programs 
affirmatively. Across all countries and on every dimension we found that Step by 
Step children perform as well or in some cases exceed the performance of children 
in traditional programs. Overall, the academic benefits of Step by Step are most 
evident in the realm of mathematics, with significant effects favoring Step by Step 
in three of the four countries. Findings favoring Step by Step might reflect the 
emphasis on exploration in these classrooms. As children experiment with objects 
and quantities in different activity centers, they have opportunities to construct 
notions of relative quantity and, when teachers join them in their explorations, 
there are many occasions when discourse about mathematical concepts can occur. 
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Some differences favoring Step by Step were also seen in literacy learning 
including receptive language. At first glance these findings are somewhat 
surprising since formal literacy instruction is emphasized in traditional programs. 
However, when taking a closer look at the nature of teacher-child interaction in 
Step by Step classrooms with their extended conversations, daily book readings, 
and writing activities, the findings are more understandable. 

In general, our assessments of creativity did not yield significant differences 
between children in Step by Step programs and children in traditional programs, 
except in the Unusual Uses activity. Results indicated greater creativity among 
Step by Step children, particularly in their ability to think in flexible ways. Thus, 
on balance, there is evidence that Step by Step classrooms are more effectively 
nurturing children’s creativity. 

Finally, when we inspect frequency distributions for children from the two types 
of programs, we consistently find that Step by Step programs seem to provide 
greater support to children who enter with less well-developed academic skills. 
This finding suggests that the Step by Step’s child-centered approach enables 
teachers to implement a program that is responsive to children’s individual needs 
rather than one dictated by a set curriculum. Since one of the critical components 
of Step by Step is individualizing the curriculum for children, our data suggests 
that teachers are making such curricular adjustments skillfully. 
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